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1 APPARATUS AND METHOD FOR IMPROVING THE QUALITY OF VIDEO 

2 COMMUNICATION OVER A PACKET-BASED NETWORK 

3 

4 CROSSREFERENCE TO RELATED APPLICATIONS 

5 [0001] This application claims priority benefit of U.S. Provisional Patent Application 

6 60/243,886, filed October 27, 2000, having the same title and inventive entity as the 

7 present application. 
8 

9 BACKGROUND 

1 0 Field of the Invention 

1 1 [0002] The field of the invention is generally video communication and more 

12 specifically improved video communication quality over a packet-based network such 

13 as an Ethernet or Internet Protocol (IP) using H. 323 protocol or similar standard. 
14 

1 5 Background Art 

16 [0003] Conventionally, a packet-based network such as an Ethernet or IP does not 

1 7 offer an end-to-end connection with a guarantee that all packets will reach their 

1 8 destination. Therefore communication over a packet-based network using different 

19 technologies is necessary to overcome packet loss. For example, communication that 

20 is not performed in real time, such as e-mail, uses protocols such as Transmission 

21 Control Protocol (TCP), which is a handshake protocol that verifies the arrival of all 

22 packets. If a packet is not received, a request is made for a retransmission of the lost 

23 packet. Alternatively, other protocols, such as User Datagram Protocol (UDP)/IP, 

24 typically send a notification about the problem of the packet loss and then handle the 

25 lost packet separately. 

1 



1 [0004] Video communications are performed in real time. However, any delay 

2 between endpoints (a physical location or apparatus that can generate and/or terminate 

3 information streams) reduces the quality of the communication performed in real 

4 time. Furthermore, a handshake protocol such as TCP/IP increases the delay, which is 

5 the time interval between the action of one side and the response of the other side. 

6 The increased delay damages the flow of a conversation. Consequently, TCP/IP is a 

7 difficult protocol to use for real time communications. 

8 [0005] Therefore, video communications use UDP/IP with H.323, Session Initiation 

9 Protocol (SIP), or similar standards instead of using TCP/IP. UDP/IP is used 

10 primarily for broadcasting video streams over a network and it is a connectionless 

1 1 protocol, which runs on top of IP networks. Unlike TCP/IP, UDP/IP has very few 

12 error recovery services. Instead UDP/IP sends and receives datagrams over an IP 

1 3 network directly. Since UDP/IP does not have a feedback mechanism the packets that 

14 do not reach their destination will be lost. The percentage of packets lost during 

1 5 transmission is typically up to about 10%. 

16 [0006] Conventional video communications use compression standards such as 

1 7 H.261 , H.263 and Moving Pictures Experts Group (MPEG). These video 

1 8 compression standards typically have two types of frames: the "Intra" frame, which is 

19 a non-referential image, and the "Non intra" frame, which is a referential frame, 

20 similar to but not limited to frames such as Inter frame (P frame), B frame, and PB 

2 1 frame. Video compression standards use the difference between the current "Non 

22 Intra" frame and a previous frame for compression, regardless whether the previous 

23 frame was an Intra or "Non Intra" frame. 

24 [0007] A video packet includes video parts, which may be a whole frame or part of a 

25 frame. Each frame may include one or more parts, where a part of a frame may 
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1 include, for example, a few Groups Of Blocks (GOB), part of a GOB, Slice, part of a 

2 Slice or a few Macro Blocks (MBs). More information about said GOBs, Slices and 

3 MBs can be found in standards H.261 , H.263, MPEG, etc. 

4 [0008] Since video compression (i.e., encoding/decoding) is based on referential 

5 information, "Non Intra" frames, the loss of a packet may damage video quality for all 

6 later frames until a new Intra frame arrives. The loss of the packet may also cause 

7 loss of video synchronization. Typically, a video decoder of an endpoint handles lost 

8 packets by requesting transmission of an Intra frame so that the referential 

9 information of the Infra frame can be used to correct the synchronization and be the 

10 basis for calculating the referential frames that follow. This request for transmission 

1 1 can be performed by using a Video Update Picture Indication (VUPI) request. 

12 [0009] A Multipoint Control Unit (MCU) is a node on a network, which provides the 

1 3 capability for two or more terminals to participate in a multimedia communication 

14 session. Therefore, the MCU may transfer a received data stream with missing 

1 5 packets, to multiple participants' endpoints. Thus the use of MCUs may create a 

1 6 bigger problem than just a lost packet in an endpoint-to-endpoint direct connection, 

17 because the lost packet will be missing to multiple endpoints rather than just one 

1 8 endpoint. . There are few known ways by which an MCU handles missing packets; 

19 none of which replace the missing packets. For example, the MCU may send video 

20 source packets while ignoring the missing packets, and let the endpoints make a VUPI 

21 request. In a videoconference with more than two endpoints, several participants may 

22 simultaneously receive communications from one source, such as a conference 

23 speaker. If a packet is lost during transmission from the video source (the conference 

24 speaker) to the MCU, each of the participants would request a VUPI from the MCU. 

25 However, the numerous VUPI requests and the INTRA frames that the follow VUPI 
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1 requests could overwhelm the network, especially if there is a high packet loss rate. 

2 Further, during Internet communications each participant may experience packet loss 

3 at a different time, and will therefore request a VUPI at different times, thereby 

4 overwhelming the network with sporadic transmissions of Intra frames. Since Intra 

5 frames are approximately 10 times "bigger" than "non Intra" frames, transmission of 

6 Intra frames may cause frame rate derogation and reduces the video quality of the 

7 conference. 

8 [0010] Therefore, there is a need for a system and method for improving video 

9 communications quality over a packet-based network. 
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1 SUMMARY OF THE INVENTION 

2 [0011] The present invention provides a novel method and apparatus for repairing 

3 video stream errors such as missing packets of an incoming compressed video stream. 

4 The system identifies a missing part or unit of a video stream (e.g., a lost packet) by 

5 tracking a sequence number associated with each packet. An analysis is then 

6 performed to determine which sub-units (e.g. GOBs, Slices, and/or MBs) are absent 

7 from the video stream, and to create one or more substitute video stream parts (i.e. 

8 repair packets) to compensate for absent sub-units. In an exemplary embodiment the 

9 repair packets include the minimum required video bits to keep the stream or flow of 

10 the video stream synchronized. This minimum bit requirement means that none of the 

1 1 MacroBlocks' (MBs') video data is placed or reconstructed in the packet. Instead, a 

12 decoder treats a repaired video stream part, such as a GOB, as an uncoded part that 

13 represents the same information as in the previous frame. The repaired information is 

14 then transferred to its destination and replaces the original information. 

15 [0012] Additionally, the MCU may ask for an "update request" from the video 

16 source, which may include a VUPI request. When compression standard allows, only 

1 7 a part of a picture that includes the missing sub-units (such as MBs) is requested. An 

1 8 example of a compression standard that allows for the request of a partial picture is 

19 H.263. 

20 [0013] Replacement of the missing parts provides for continuity of the video stream 

21 at a receiver site. This continuity enables smooth parsing, without synchronization 

22 losses and prevents re-synchronization process by the video "decoders." The decoder, 

23 among other places, may be located inside the MCU, or outside of the MCU at the 

24 participant's endpoint. 
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1 [0014] Alternatively, if the missing part belongs to a nonreferential image (e.g., an 

2 Intra frame), then the video stream is not corrected. Instead a new nonreferential 

3 image (such as Intra frame) is requested. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

1 [0015] The invention will be more readily understood from reading the following 

2 specification and by reference to the accompany drawings showing an example of the 

3 invention. 

4 [0016] FIG. 1 is a block diagram of a prior art network interface unit; 

5 [0017] FIG. 2 is a block diagram of an exemplary embodiment of the present 

6 invention; and 

7 [0018] FIG. 3 is a flow diagram of the steps of an exemplary embodiment of the 

8 present invention. 
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1 DETAILED DESCRIPTION OF THE INVENTION 

2 [0019] Fig. 1 is a block diagram of a prior art network interface unit 100. As shown, 

3 a Local Area Network (LAN) controller 1 10 grabs required multimedia packets from 

4 a LAN 1 01 via a buffer 111. Preferably, the LAN controller 1 10 then sorts the 

5 multimedia packets into two types of streams: control streams 160 and media streams 

6 135. The control stream 160 is transferred via buffers 161 to a backplane 170. 

7 [0020] The media streams 135 are transferred to a Real Time Transfer Protocol (RTP) 

8 unit 120, a packetizing/depacketizing unit that sorts the streams into Audio streams 

9 150, Video streams 140 and Data streams 130 and transfers them via buffers 141, 151 

10 and 131, respectively, to the backplane 170. More specifically, video streams 140 are 

1 1 transferred through buffer 141 to the backplane 170 without repairing the missing 

1 2 packets, while audio streams 1 50 are transferred via buffer 151. 

1 3 [0021] Conversely, FIG. 2 is a block diagram illustrating an exemplary embodiment 

14 of a network interface unit 200 according to the present invention. Preferably, the 

1 5 network interface logical unit 200 includes a LAN controller 1 10, RTP unit 120, 

16 Video streams 140, Missing Packets Repair logical Unit (MPRU) 210, Packet 

1 7 Analyzer logical unit (PA) 220, memory 230, GOB history memory 240, GOB 

1 8 Analyzer logical unit (GA) 250, Packet Repair logical unit (PR) 260, and Temporary 

1 9 Packets Memory (TPM) 265 . 

20 [0022] It should be noted that the term "GOB" represents a "Slice" for situations 

21 where slices are used instead of GOBs. In such a case, the GOB ID Number is 

22 replaced by the Slice ID, which is the MBA field in the Slice header. The MBA 

23 indicates the first MB in the Slice. 

24 [0023] The network interface logical unit 200 is similar to network interface unit 100 

25 (FIG. 1) in that they both include a LAN controller 1 10 and RTP unit 120. However, 
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1 in contrast to the prior art device of network interface 100, the FIG. 2 system also 

2 includes a Missing Packets Repair logical Unit (MPRU) 210. 

3 [0024] Operationally, video streams 140 from the RTP 120 are transferred to the 

4 MPRU 210. Inside the MPRU 210 the video stream 140 is distributed to three logical 

5 units: PA 220, GA 250, and PR 260. The PA 220 reads a current packet sequence 

6 number of a received packet, compares the current sequence number to the expected 

7 sequence number, and in case of mismatch sends indications of packet loss to the 

8 LAN controller 1 10, GA 250, and PR 260. The expected sequence number is based 

9 on the history of the received packets stored in its memory 230. 

10 [0025] The GA 250 analyzes which GOBs/Slices are associated with each received 

1 1 packet and stores this information in the GOB history memory 240. The analyzing 

1 2 can be done by parsing the video stream of a packet and/or by grabbing the 

1 3 information regarding the GOB/Slice ID from the RTP header. This information 

14 about the GOB/Slice header in the RTP Header does not always exist and is based on 

15 the RFC. 

1 6 [0026] When the GA 250 receives an indication that there are missing packets prior to 

1 7 the current packet, GA 250 uses the last stored information in the memory 240 and the 

1 8 first GOB/Slice ID Number that was found in the current packet to analyze which 

1 9 GOBs/MBs are missing. The GA 250 then sends this information to the PR 260, 

20 which in turn receives the incoming video stream 140 of packets and stores them in 

2 1 the Temporary Packet Memory (TPM) 265. Thus, the TPM 265 receives both 

22 indications about a missing packet from the PA 220 and the identification numbers of 

23 the corresponding missing GOB/MBs from the GA 250. If there are no missing 

24 packets, the PR 260 reads the last stored packet from the TPM 265 and sends the 

25 stored packet, as is, via buffer 271 to the backplane 170. When the PR 260 receives 
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1 the indications from the PA 220 and from GA 250, the PR 260 creates repaired 

2 packets and sends the corrected stream, including the repaired packets with the stored 

3 packets from the TPM 265 via buffer 271 to the backplane 170. The determination of 

4 which GOBs/MBs are missing is based on the compression standard as explained 

5 below by example. 

6 [0027J Fig. 3 is a flow diagram illustrating steps of an exemplary method for 

7 repairing missing packets according to the present invention. Initially, the MPRU 210 

8 (FIG. 2) receives a next packet in step 300. The PA 220 (FIG. 2) then reads the 

9 sequence number of the packet in step 310 and compares the sequence number of the 

10 packet to the sequence number of the previous packet stored in the memory 230 (FIG. 

1 1 2). Subsequently, the PA 220 (FIG. 2) checks if the packet is consecutive in decision 

12 box 320 based on the comparison of the sequence numbers. If the packets are 

13 consecutive, the PA 220 stores the current packet's sequence number in the memory 

14 230 and returns to step 300 to receive a new packet. However, if the packets are not 

15 consecutive, meaning that one or more packets have been lost, then PA 220 checks if 

16 the missing packet belongs to an Intra frame or "Non Intra" frame in step 340. 

17 [0028] One way of keeping track of the type of frame is to sample the indication of 

1 8 frame type. For example, the PA 220 may check the frame type by reading a "frame's 

1 9 header," GOB "0" of the last packet. In H.263, the frame header is part of a header of 

20 GOB "0" while in H.261 the frame header is identified by GOB ID Number "0". 

21 [00291 If the missing packet is part of Intra frame 340, no packet repair is necessary 

22 and a VXJPI request is performed in step 380 and the process returns to step 300 to 

23 receive the next packet. If the packet contains part of a "Non Intra" frame the PA 220 

24 sends a missing packets acknowledgement to the GA 250 (FIG. 2) and the PR 260 

25 (FIG. 2). The GA 250 in step 350 uses the information stored in the memory 240 
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1 (FIG. 2) to get the last GOB ID Number that arrived before the missing packets. For 

2 example, if the last GOB before the missing packet is GOB (j), then the GA 250 finds 

3 the first GOB in the current packet in step 360, e.g., GOB (k). Based on these two 

4 GOB ID Numbers (i.e., (j) and (k)), the compression standard, and the source format 

5 (e.g., Common Intermediate Format (CIF), 4 CIF, or Quarter CIF (QCIF)), the GA 

6 250 determines in step 365 which are the missing GOBs. 

7 [0030] In the case that the compression stream is using Slices instead of GOBs, the 

8 GA 250 analyzes and stores the last Slice ID, which is the MBA field in the Slice 

9 header of the last received packet. Then the GA finds the Slice ID of the first Slice. 

10 By using this two MB As and the width of the Slice the GA 250 analyzes which MBs 

1 1 are the missing. 

12 [0031] The following are a few examples of the analysis, for video algorithms H.261 

13 and H.263 and two sources formats CIF and QCIF. The numbers in the following 

14 table specify the sequence of GOB ID Numbers used in those cases. 





QCIF 


CIF 


H.261 


0,1,3,5 


0,1,2. ..12 


H.263 


0.1.2...8 


0,1...16,17 



15 
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1 [0032] The GA 250 recognizes the missing GOBs by finding, the GOBs missing in 

2 the sequence. Thus: 



3 a. For H.26 1 and QCDF, if j=0 and k=5, then the repaired packet 

4 would include GOBs 1 and 3. 

5 b. For H.26 1 and CIF, if j=4 and k=0, then the repaired packet would 

6 include GOBs 5,6,7... 12. 

7 c. For H.263 and QCIF, if j=6 and k-2, there are 2 packets that need 

8 to be repaired. The first packet includes GOBs 7 and 8, and the second 

9 packet includes GOBs 0 and 1 . A packet is not allowed to include GOBs 

1 0 from different frames. 

1 1 d. For H.263 and CIF, if j= 1 7 and k=6, the repair packet includes 

12 GOBsO, 1,2. ..5. 

1 3 e. For H.26 1 and CIF, if j=4 and k=5 no repaired packet is needed. 

14 If a GOB ID Number is not found in the current packet, no analysis can be 

1 5 performed and the GA 250 drops the current packet. Subsequently, the 

16 process returns to step 300 and waits for the next packet. 



1 7 [0033] The GA 250 transfers the results of its analysis (i.e., the missing GOB ID 

1 8 Number) to logical unit PR 260 (FIG. 2), which in turn prepares the necessary 

19 repaired packet or packets in step 370. 

20 [0034] The packet repair is done based on the compression standard, and the source 

21 format, each packet includes the necessary bits plus some extra bits or "stuffing" that 

22 keep the continuity of the stream as required by the protocol in use. For example 

23 H.261 requires just the GOB headers with no byte alignment, while H.263 requires 1 

24 bit for each non-coded Macro Block (MB). Since H.263 CIF GOB includes 22 MBs, 

25 22 bits of T are needed. H.263 QCIF GOB includes only 1 1 MBs, so in this case 1 1 
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1 bits of ' r are needed. These bits are added to the GOB header. Further, the H.263 

2 frame should be byte aligned, which is guaranteed by the H.263 stuffing. 

3 [0035] The PR 260 sends the repaired packet or packets with the current packet as a 

4 corrected video stream to the backplane in step 390, and sends an update request to 

5 the video source in step 380. Conversely, if the PR 260 does not get a missing packet 

6 indication from the PA 220, the PR 260 sends the current packet, as is, to buffer 271. 

7 The update request might be a VUPI request, however it is not always necessary. 

8 Alternatively, in H.263 a message "VideoNotDecodedMb" may be used and only the 

9 missing GOBs/MBs may be updated. 

10 [0036] The MPRU 210 can be implemented by software (e.g., an additional software 

1 1 package loaded to an existing processor, such as RTP processor 1 20). Alternatively, 

1 2 the MPRU 2 1 0 may be an additional processor including programs or special 

1 3 hardware for all three units, the PA 220, the GA 250 and the PR 260, or three separate 

14 processors, one per unit or any combination of one to three processors and/or 

1 5 hardware units. Memories 230, 240 and TPM 265 (FIG. 2) can be implemented by 

16 any one of, or any combination of, SRAM, DRAM, SDRAM, internal and/or external 

17 memory. 

18 [0037] Although the above exemplary embodiments are given in terms of streams, 

19 packets, GOBs, and Slices, the invention can be applied to video streams that are 

20 divided into other units, parts, and/or sub-units. Similarly, the missing packet repair 

21 unit, MPRU 210, is just an example of a missing part repair unit. 

22 [0038] As can be understood from the above description, the present system improves 

23 traffic over a network by replacing missing GOBs/Slices, and thereby enhances the 

24 continuity of data streams at a receiver side. The continuity of the data stream enables 

25 smooth parsing (without synchronization losses and re-synchronization process) by 
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1 video "decoders" and improves the quality of video communication. In the 

2 description and claims of the present application, each of the verbs, "comprise" 

3 "include" and "have", and conjugates thereof, are used to indicate that the object or 

4 objects of the verb are not necessarily a complete listing of members, components, 

5 elements or parts of the subject or subjects of the verb. 

6 [0039] While the invention has been described with reference to specific 

7 embodiments, it will be understood by those skilled in the art that various changes 

8 may be made and equivalents may be substituted for elements thereof without . 

9 departing from the true spirit and scope of the invention. In addition, modifications 
10 may be made without departing from the essential teachings of the invention. 
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WHAT IS CLAIMED 

1. An apparatus for facilitating multimedia communication between a plurality of 
endpoints over a packet based network, each respective endpoint sending a compressed 
video output signal and receiving a compressed video input signal, having at least one 
network interface to a packet-based network, the network interface comprising: 

a missing packets repair logical unit, said missing packets repair logical unit 
handling missing packets and thereby maintaining continuity of a video stream and 
reducing traffic over said network. 

2. The network interface of claim 1, wherein the missing packets repair logical unit 
further comprises: 

a first analyzer that analyzes if a packet is missing; 

a second analyzer that analyzes which Group Of Blocks (GOBs) are in the 
missing packet; and 

a repair unit that replaces the missing packets. 

3. The apparatus of claim 1 wherein the network is a local area network. 

4. The apparatus of claim 1 wherein the network is a wide area network. 
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1 5. The apparatus of claim 1 wherein the video stream is repaired during a video stream 

2 receiving. 

1 6. A system comprising: 

2 a network interface unit having 

3 a video stream repair unit that receives a video stream from a real time 

4 protocol unit, having 

5 a detector unit that detects missing packets, 

6 an analyzer unit that analyzes which video parts are missing, 

7 a replacement unit that 

8 receives 

9 an indication from the detector unit that packets are 

10 missing, and 

1 1 information from the analyzer unit, the information 

12 including which video parts are missing, and 

13 in response to receiving the indication and the information 

14 replaces a missing packets in a video stream during transmission of 

15 the video stream over a network as part of a receive process. 
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1 7. A method for repairing missing packets in video communication over a packet-based 

2 network, the method comprising: 

3 analyzing if at least one packet is missing; 

4 determining which Group of Blocks (GOBs) are missing; 

5 preparing new packets which will replace the GOBs; and 

6 sending the new packets to a destination. 

1 8. The method of claim 7 wherein the destination is a point remote from where the 



2 analyzing occurs. 

1 9. A method comprising: 

2 transmitting a video session over a network; and 

3 replacing a missing part from the video session while the video session is being 

4 transmitted. 

1 10. The method of claim 9 wherein the missing part is replaced in an intermediate node 

2 during a receiving process that receives the video session. 

1 11. The method of claim 10 further comprising detecting the missing part during the 

2 receiving process. 
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1 12. The method of claim 1 1 wherein detecting the missing part includes detecting if a 

2 part of the video session received is out of sequence. 

1 13. The method of claim 9 wherein replacing the missing part is dependent on the frame 

2 type. 

1 14. The method of claim 12 wherein: 

2 each part of a video session has an ID number; and 

3 analyzing which part is missing comprises 

4 finding the ID number of the last received video session part , and 

5 finding the ID number of a first video session part of a stream currently 

6 being analyzed. 

1 15. The method of claim 9 wherein the missing part is at least one GOB. 

1 16. A computer usable storage medium having stored thereon a method comprising: 

2 transmitting a video session over a network; and 

3 replacing a missing part from the video session while the video session is being 

4 transmitted. 

1 17. The computer usable storage medium of claim 16 wherein the missing part is 

2 replaced in an intermediate node during a receiving process that receives the video 

3 session. 
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1 18. The computer usable storage medium of claim 17, the method further comprising 

2 detecting the missing part during the receiving process. 

1 19. The computer usable storage medium of claim 18 wherein detecting the missing part 

2 includes detecting if a part of the video session received is out of sequence. 

1 20. The computer usable storage medium of claim 16 wherein replacing the missing part 

2 is dependant on a frame type. 



1 21. The computer usable storage medium of claim 1 9 wherein: 

2 each part of a video session has an ID number; and 

3 analyzing which part is missing comprises; 

4 finding the ID number of the last received video session part , and 

5 finding the ID number of a first video session part of a stream currently 

6 being analyzed. 

1 22. The computer usable medium of claim 16 wherein the missing part is a packet. 

1 23. The network interface of claim 1, wherein the missing packets repair logical unit 

2 further comprises: 

3 a first analyzer that analyzes if a packet is missing; 
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a second analyzer that analyzes which Slices are in the missing packet; and 
a repair unit that replaces the missing packets. 

24. A method for repairing missing packets in video communication over a packet-based 
network, the method comprising: 

analyzing if at least one packet is missing; 
determining which Slices are missing; 
preparing new packets which will replace the Slices; and 
sending the new packets to a destination. 

25. The method of claim 9 wherein the missing part is at least one Slice. 
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ABSTRACT OF THE INVENTION 
A network interface unit is provided with a missing packets repair logical unit, 
which repairs real time video transmissions by replacing the missing packets. The 
missing packets repair unit detects and replaces missing packets in a received video 
stream. The missing packets can be detected by finding packets that are out of sequence. 
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14 BACKGROUND 

15 Field of the Invention 

16 Fi e ld of th e Invention 

17 [0002] The field of the invention is generally video communication and more 

1 8 specifically improved video communication quality over a packet-based network such 

19 as an Ethernet or Internet Protocol (IP) using H.323 protocol or similar standard. 
20 

21 Backg round Art 

22 Background Art 

23 (0003) Conventionally, a packet-based network such as an Ethernet or IP does 

24 not offer an end-to-end connection with a guarantee that all packets will reach their 

25 destination. Therefore communication over a packet-based network using different 
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1 technologies is necessary to overcome packet loss. For example, communication that 

2 is not performed in real time, such as e-mail, uses protocols such as Transmission 

3 Control Protocol (TCP), which is a handshake protocol that verifies the arrival of all 

4 packets. If a packet is not received, a request is made for a retransmission of the lost 

5 packet. Alternatively, other protocols, such as User Datagram Protocol (UDP)/IP, 

6 typically send a notification about the problem of the packet loss and then handle the 

7 lost packet separately. 

8 [0004] Video communications are performed in real time. However, any 

9 delay between endpoints (a physical location or apparatus that can generate and/or 

10 terminate information streams) reduces the quality of the communication performed 

1 1 in real time. Furthermore, a handshake protocol such as TCP/IP increases the delay, 

12 which is the time interval between the action of one side and the response of the other 

1 3 side. The increased delay damages the flow of a conversation. Consequently, TCP/IP 

14 is a difficult protocol to use for real time communications. 

1 5 [0005] Therefore, video communications use UDP/IP with H.323, Session 

1 6 Initiation Protocol (SIP), or similar standards instead of using TCP/IP. UDP/IP is 

1 7 used primarily for broadcasting video streams over a network and it is a 

1 8 connectionless protocol, which runs on top of IP networks. Unlike TCP/IP, UDP/IP 

19 has very few error recovery services. Instead UDP/IP sends and receives datagrams 

20 over an IP network directly. Since UDP/IP does not have a feedback mechanism the 

21 packets that do not reach their destination will be lost. The percentage of packets lost 

22 during transmission is typically up to about 10%. 

23 [0006] Conventional video communications use compression standards such as 

24 H.261, H.263 and Moving Pictures Experts Group (MPEG). These video 

25 compression standards typically have two types of frames: the "Intra" frame, which is 
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1 a non-referential image, and the "Non intra" frame, which is a referential frame, 

2 similar to but not limited to frames such as Inter frame (P frame), B frame, and PB 

3 frame. Video compression standards use the difference between the current "Non 

4 Intra" frame and a previous frame for compression, regardless whether the previous 

5 frame was an Intra or "Non Intra" frame. 

6 [0007] A video packet includes video parts, which may be a whole frame or 

7 part of a frame. Each frame may include one or more parts, where a part of a frame 

8 may include, for example, a few Groups Of Blocks (GOB), part of a GOB, Slice, part 

9 of a Slice or a few Macro Blocks (MBs). More information about said GOBs, Slices 

1 0 and MBs can be found in standards H.26 1 , H.263, MPEG, etc. 

1 1 [0008] Since video compression (i.e., encoding/decoding) is based on 

12 referential information, "Non Intra" frames, the loss of a packet may damage video 

13 quality for all later frames until a new Intra frame arrives. The loss of the packet may 

14 also cause loss of video synchronization. Typically, a video decoder of an endpoint 

1 5 handles lost packets by requesting transmission of an Intra frame so that the 

16 referential information of the Infra frame can be used to correct the synchronization 

17 and be the basis for calculating the referential frames that follow. This request for 

1 8 transmission can be performed by using a Video Update Picture Indication (VUPI) 

19 request. 

20 [0009] A Multipoint Control Unit (MCU) is a node on a network, which 

21 provides the capability for two or more terminals to participate in a multimedia 

22 communication session. Therefore, the MCU may transfer a received data stream 

23 with missing packets, to multiple participants' endpoints. Thus the use of MCUs 

24 may create a bigger problem than just a lost packet in an endpoint-to-endpoint direct 

25 connection, because the lost packet will be missing to multiple endpoints rather than 
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1 just one endpoint. . There are few known ways by which an MCU handles missing 

2 packets; none of which replace the missing packets. For example, the MCU may send 

3 video source packets while ignoring the missing packets, and let the endpoints make a 

4 VUPI request. In a videoconference with more than two endpoints, several 

5 participants may simultaneously receive communications from one source, such as a 

6 conference speaker. If a packet is lost during transmission from the video source (the 

7 conference speaker) to the MCU, each of the participants would request a VUPI from 

8 the MCU. However, the numerous VUPI requests and the INTRA frames that the 

9 follow VUPI requests could overwhelm the network, especially if there is a high 

10 packet loss rate. Further, during Internet communications each participant may 

1 1 experience packet loss at a different time, and will therefore request a VUPI at 

12 different times, thereby overwhelming the network with sporadic transmissions of 

13 Intra frames. Since Intra frames are approximately 10 times "bigger" than "non Intra" 

14 frames, transmission of Intra frames may cause frame rate derogation and reduces the 

1 5 video quality of the conference. 

1 6 [0010] Therefore, there is a need for a system and method for improving video 

1 7 communications quality over a packet-based network. 
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1 SUMMARY OF THE INVENTION 

2 [001 1] The present invention provides a novel method and apparatus for repairing 

3 video stream errors such as missing packets of an incoming compressed video stream. 

4 The system identifies a missing part or unit of a video stream (e.g., a lost packet) by 

5 tracking a sequence number associated with each packet. An analysis is then 

6 performed to determine which sub-units (e.g. GOBs, Slices, and/or MBs) are absent 

7 from the video stream, and to create one or more substitute video stream parts (i.e. 

8 repair packets) to compensate for absent sub-units. In an exemplary embodiment the 

9 repair packets include the minimum required video bits to keep the stream or flow of 

10 the video stream synchronized. This minimum bit requirement means that none of the 

1 1 MacroBlocks' (MBs') video data is placed or reconstructed in the packet. Instead, a 

12 decoder treats a repaired video stream part, such as a GOB, as an uncoded part that 

13 represents the same information as in the previous frame. The repaired information is 

14 then transferred to its destination and replaces the original information. 

15 [0012] Additionally, the MCU may ask for an "update request" from the video 

16 source, which may include a VUPI request. When compression standard allows, only 

1 7 a part of a picture that includes the missing sub-units (such as MBs) is requested. An 

1 8 example of a compression standard that allows for the request of a partial picture is 

19 H.263. 

20 [0013] Replacement of the missing parts provides for continuity of the video stream 

21 at a receiver site. This continuity enables smooth parsing, without synchronization 

22 losses and prevents re-synchronization process by the video "decoders." The decoder, 

23 among other places, may be located inside the MCU, or outside of the MCU at the 

24 participant's endpoint. 
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1 [0014] Alternatively, if the missing part belongs to a nonreferential image (e.g., an 

2 Intra frame), then the video stream is not corrected. Instead a new nonreferential 

3 image (such as Intra frame) is requested. 
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1 BRIEF DESCRIPTION OF THE DRAWINGS 

2 [0015] The invention will be more readily understood from reading the following 

3 specification and by reference to the accompany drawings showing an example of the 

4 invention. 

5 [0016] FIG. 1 is a block diagram of a prior art network interface unit; 

6 [0017] FIG. 2 is a block diagram of an exemplary embodiment of the present 

7 invention; and 

8 [0018] FIG. 3 is a flow diagram of the steps of an exemplary embodiment of the 

9 present invention. 
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DETAIL DESCRIPTION OF THE INVENTION 
[0019] Fig. 1 is a block diagram of a prior art network interface unit 100. As shown, 
a Local Area Network (LAN) controller 110 grabs required multimedia packets from 
a LAN 101 via a buffer 111. Preferably, the LAN controller 1 1 0 then sorts the 
multimedia packets into two types of streams: control streams 160 and media streams 
135. The control stream 160 is transferred via buffers 161 to a backplane 170. 
[0020] The media streams 135 are transferred to a Real Time Transfer Protocol (RTP) 
unit 120, a packetizing/depacketizing unit that sorts the streams into Audio streams 
150, Video streams 140 and Data streams 130 and transfers them via buffers 141,151 
and 131, respectively, to the backplane 170. More specifically, video streams 140 are 
transferred through buffer 141 to the backplane 170 without repairing the missing 
packets, while audio streams 150 are transferred via buffer 151. 
[0021] Conversely, FIG. 2 is a block diagram illustrating an exemplary embodiment 
of a network interface unit 200 according to the present invention. Preferably, the 
network interface logical unit 200 includes a LAN controller 1 10, RTP unit 120, 
Video streams 140, Missing Packets Repair logical Unit (MPRU) 210, Packet 
Analyzer logical unit (PA) 220, memory 230, GOB history memory 240, GOB 
Analyzer logical unit (GA) 250, Packet Repair logical unit (PR) 260, and Temporary 
Packets Memory (TPM) 265. 

[0022] It should be noted that the term "GOB" represents a "Slice" for situations 
where slices are used instead of GOBs. In such a case, the GOB ID Number is 
replaced by the Slice ID, which is the MBA field in the Slice header. The MBA 
indicates the first MB in the Slice. 
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1 [0023) The network interface logical unit 200 is similar to network interface unit 100 

2 (FIG. 1 ) in that they both include a LAN controller 1 10 and RTP unit 120. However, 

3 in contrast to the prior art device of network interface 100, the FIG. 2 system also 

4 includes a Missing Packets Repair logical Unit (MPRU) 210. 

5 [0024] Operationally, video streams 140 from the RTP 120 are transferred to the 

6 MPRU 210. Inside the MPRU 210 the video stream 140 is distributed to three logical 

7 units: PA 220, GA 250, and PR 260. The PA 220 reads a current packet sequence 

8 number of a received packet, compares the current sequence number to the expected 

9 sequence number, and in case of mismatch sends indications of packet loss to the 

10 LAN controller 1 10, GA 250, and PR 260. The expected sequence number is based 

11 on the history of the received packets stored in its memory 230. 

12 [0025] The GA 250 analyzes which GOBs/Slices are associated with each received 

13 packet and stores this information in the GOB history memory 240. The analyzing 

14 can be done by parsing the video stream of a packet and/or by grabbing the 

15 information regarding the GOB/^^Slice_ID from the RTP header. This information 

16 about the GOB/^^Slice header in the RTP Header does not always exist and is 

1 7 based on the J1||REC. 

18 [0026] When the GA 250 receives an indication that there are missing packets prior to 

19 the current packet, GA 250 uses the last stored information in the memory 240 and the 

20 first GOB/Slice ID Number that was found in the current packet to analyze which 

21 GOBs/MBs are missing. The GA 250 then sends this information to the PR 260, 

22 which in turn receives the incoming video stream 140 of packets and stores them in 

23 the Temporary Packet Memory (TPM) 265. Thus, the TPM 265 receives both 

24 indications about a missing packet from the PA 220 and the identification numbers of 

25 the corresponding missing GOB/MBs from the GA 250. If there are no missing 
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1 packets, the PR 260 reads the last stored packet from the TPM 265 and sends the 

2 stored packet, as is, via buffer 271 to the backplane 170. When the PR 260 receives 

3 the indications from the PA 220 and from GA 250, the PR 260 creates repaired 

4 packets and sends the corrected stream, including the repaired packets with the stored 

5 packets from the TPM 265 via buffer 271 to the backplane 170. The determination of 

6 which GOBs/MBs are missing is based on the compression standard as explained 

7 below by example. 

8 [0027] Fig. 3 is a flow diagram illustrating steps of an exemplary method for 

9 repairing missing packets according to the present invention. Initially, the MPRU 210 

10 (FIG. 2) receives a next packet in step 300. The PA 220 (FIG. 2) then reads the 

1 1 sequence number of the packet in step 3 1 0 and compares the sequence number of the 

12 packet to the sequence number of the previous packet stored in the memory 230 (FIG. 

13 2). Subsequently, the PA 220 (FIG. 2) checks if the packet is consecutive in decision 

14 box 320 based on the comparison of the sequence numbers. If the packets are 

15 consecutive, the PA 220 stores the current packet's sequence number in the memory 

16 230 and returns to step 300 to receive a new packet. However, if the packets are not 

17 consecutive, meaning that one or more packets have been lost, then PA 220 checks if 

1 8 the missing packet belongs to an Intra frame or "Non Intra" frame in step 340. 

19 [0028] One way of keeping track of the type of frame is to sample the indication of 

20 frame type. For example, the PA 220 may check the frame type by reading a "frame's 

2 1 header," GOB "0" of the last packet. In H.263, the frame header is part of a header of 

22 GOB "0" while in H.261 the frame header is identified by GOB ID Number "0". 

23 [0029] If the missing packet is part of Intra frame 340, no packet repair is necessary 

24 and a VUPI request is performed in step 380 and the process returns to step 300 to 

25 receive the next packet. If the packet contains part of a u Non Intra" frame the PA 220 
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1 sends a missing packets acknowledgement to the GA 250 (FIG. 2) and the PR 260 

2 (FIG. 2). The GA 250 in step 350 uses the information stored in the memory 240 

3 (FIG. 2) to get the last GOB ID Number that arrived before the missing packets. For 

4 example, if the last GOB before the missing packet is GOB (j), then the GA 250 finds 

5 the first GOB in the current packet in step 360, e.g., GOB (k). Based on these two 

6 GOB ID Numbers (i.e., (j) an d (k)), the compression standard, and the source format 

7 (e.g., Common Intermediate Format (CIF), 4 CIF, or Quarter CIF (QCIF)), the GA 

8 250 determines in step 365 which are the missing GOBs. 

9 [0030] In the case that the compression stream is using Slices instead of GOBs, the 

10 GA 250 analyzes and stores the last Slice ID, which is the MBA field in the Slice 

1 1 header of the last received packet. Then the GA finds the Slice ID of the first Slice. 

12 By using this two MB As and the width of the Slice the GA 250 analyzes which MBs 

13 are the missing. 

14 [0031] The following are a few examples of the analysis, for video algorithms H.261 

1 5 and H.263 and two sources formats CIF and QCIF. The numbers in the following 

1 6 table specify the sequence of GOB ID Numbers used in those cases. 
17 





QCIF 


CIF 


H.261 


0,1,3,5 


0,1,2. ..12 


H.263 


0,1,2...8 


0,1...16,17 



18 
19 

20 [0032] The GA 250 recognizes the missing GOBs by finding, the GOBs missing in 

21 the sequence. Thus: 
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1 a. For H.261 and QCIF, if j=0 and k=5, then the repaired packet would 

2 include GOBs 1 and 3. 

3 b. For H.261 and CIF, if j=4 and k=0, then the repaired packet would 

4 include GOBs 5,6,7... 12. 

5 c. For H.263 and QCIF, if j=6 and k=2, there are 2 packets that need to be 

6 repaired. The first packet includes GOBs 7 and 8, and the second 

7 packet includes GOBs 0 and 1. A packet is not allowed to include 

8 GOBs from different frames. 

9 d. For H.263 and CIF, if j= 17 and k=6, the repair packet includes GOBs 

10 0, 1,2.. .5. 

1 1 e. For H.26 1 and CIF, if j=4 and k=5 no repaired packet is needed. 

12 If a GOB ID Number is not found in the current packet, no analysis can be 

13 performed and the GA 250 drops the current packet. Subsequently, the process 

14 returns to step 300 and waits for the next packet. 

15 [0033] The GA 250 transfers the results of its analysis (i.e., the missing GOB ID 

16 Number) to logical unit PR 260 (FIG. 2), which in turn prepares the necessary 

1 7 repaired packet or packets in step 370. 

18 (0034] The packet repair is done based on the compression standard, and the 

19 source format, each packet includes the necessary bits plus some extra bits or 

20 "stuffing" that keep the continuity of the stream as required by the protocol in use. 

21 For example H.261 requires just the GOB headers with no byte alignment, while 

22 H.263 requires 1 bit for each non-coded Macro Block (MB). Since H.263 CIF 

23 GOB includes 22 MBs, 22 bits of 4 V are needed. H.263 QCIF GOB includes only 

24 1 1 MBs, so in this case 1 1 bits of ' T are needed. These bits are added to the GOB 
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1 header. Further, the H.263 frame should be byte aligned, which is guaranteed by 

2 the H.263 stuffing. 

3 [0035] The PR 260 sends the repaired packet or packets with the current packet as 

4 a corrected video stream to the backplane in step 390, and sends an update request 

5 to the video source in step 380. Conversely, if the PR 260 does not get a missing 

6 packet indication from the PA 220, the PR 260 sends the current packet, as is, to 

7 buffer 271 . The update request might be a VUPI request, however it is not always 

8 necessary. Alternatively, in H.263 a message "VideoNotDecodedMb" may be 

9 used and only the missing GOBs/MBs may be updated. 

10 [0036] The MPRU 210 can be implemented by software (e.g., an additional software 

1 1 package loaded to an existing processor, such as RTP processor 120). Alternatively, 

12 the MPRU 210 may be an additional processor including programs or special 

13 hardware for all three units, the PA 220, the GA 250 and the PR 260, or three separate 

14 processors, one per unit or any combination of one to three processors and/or 

15 hardware units. Memories 230, 240 and TPM 265 (FIG. 2) can be implemented by 

16 any one of, or any combination of, SRAM, DRAM, SDRAM, internal and/or external 

17 memory. 

* 18 [0037] Although the above exemplary embodiments are given in terms of streams, 

19 packets, GOBs, and Slices, the invention can be applied to video streams that are 

20 divided into other units, parts, and/or sub-units. Similarly, the missing packet repair 

2 1 unit, MPRU 2 1 0, is just an example of a missing part repair unit. 

22 [0038] As can be understood from the above description, the present system improves 

23 traffic over a network by replacing missing GOBs/Slices, and thereby enhances the 

24 continuity of data streams at a receiver side. The continuity of the data stream enables 

25 smooth parsing (without synchronization losses and re-synchronization process) by 



13 



video "decoders" and improves the quality of video communication. In the 
description and claims of the present application, each of the verbs, "comprise" 
"include" and "have", and conjugates thereof, are used to indicate that the object or 
objects of the verb are not necessarily a complete listing of members, components, 
elements or parts of the subject or subjects of the verb. 

[0039] While the invention has been described with reference to specific 
embodiments, it will be understood by those skilled in the art that various changes 
may be made and equivalents may be substituted for elements thereof without 
departing from the true spirit and scope of the invention. In addition, modifications 
may be made without departing from the essential teachings of the invention. 
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WHAT IS CLAIMED 

1 1. An apparatus for facilitating multimedia communication between a plurality of 

2 endpoints over a packet based network, each respective endpoint sending a 

3 compressed video output signal and receiving a compressed video input signal, having 

4 at least one network interface to a packet-based network, the network interface 

5 comprising: 

6 a missing packets repair logical unit, said missing packets repair logical unit 

7 handling missing packets and thereby maintaining continuity of a video stream and 

8 reducing traffic over said network. 

1 2. The network interface of claim 1, wherein the missing packets repair logical unit 

2 further comprises: 

3 a first analyzer that analyzes if a packet is missing; 

4 a second analyzer that analyzes which Group Of Blocks (GOBs) are in the 

5 missing packet; and 

6 a repair unit that replaces the missing packets. 

1 3. The apparatus of claim 1 wherein the network is a local area network. 

1 4. The apparatus of claim 1 wherein the network is a wide area network. 

1 5. The apparatus of claim 1 wherein the video stream is repaired during a video 

2 stream receiving. 
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6. A system comprising: 

a network interface unit having 

a video stream repair unit that receives a video stream from a real time 
protocol unit, having 

a detector unit that detects missing packets, 
an analyzer unit that analyzes which video parts are missing, 
a replacement unit that 
receives 

an indication from the detector unit that packets are 
missing, and 

information from the analyzer unit, the information 
including which video parts are missing, and 
in response to receiving the indication and the information 
replaces a missing packets in a video stream during transmission of 
the video stream over a network as part of a receive process. 
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7. A method for repairing missing packets in video communication over a packet-based 
network, the method comprising: 

analyzing if at least one packet is missing; 
determining which Group of Blocks (GOBs) are missing; 
preparing new packets which will replace the GOBs; and 
sending the new packets to a destination. 

8. The method of claim 7 wherein the destination is a point remote from where the 
analyzing occurs. 

9. A method comprising: 

transmitting a video session over a network; and 

replacing a missing part from the video session while the video session is being 
transmitted. 

10. The method of claim 9 wherein the missing part is replaced in an intermediate node 
during a receiving process that receives the video session. 

11. The method of claim 10 further comprising detecting the missing part during the 
receiving process. 
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1 12. The method of claim 1 1 wherein detecting the missing part includes detecting if a 

2 part of the video session received is out of sequence 

1 13. The method of claim 9 wherein replacing the missing part is dependent on the frame 

2 type. 

1 14. The method of claim 12 wherein: 

2 each part of a video session has an ID number; and 

3 analyzing which part is missing comprises 

4 finding the ID number of the last received video session part , and 

5 finding the ID number of a first video session part of a stream currently 

6 being analyzed. 

1 15. The method of claim 9 wherein the missing part is at least one GOB. 

1 16. A computer usable storage medium having stored thereon a method comprising: 

2 transmitting a video session over a network; and 

3 replacing a missing part from the video session while the video session is being 

4 transmitted. 
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1 17. The computer usable storage medium of claim 16 wherein the missing part is 

2 replaced in an intermediate node during a receiving process that receives the video 

3 session. 

1 18. The computer usable storage medium of claim 17, the method further comprising 

2 detecting the missing part during the receiving process. 

1 19. The computer usable storage medium of claim 18 wherein detecting the missing part 

2 includes detecting if a part of the video session received is out of sequence 

1 20. The computer usable storage medium of claim 16 wherein replacing the missing part 

2 is dependant on a frame type. 

1 21. The computer usable storage medium of claim 1 9 wherein: 



2 each part of a video session has an ID number; and 

3 analyzing which part is missing comprises; 

4 finding the ID number of the last received video session part , and 

5 finding the ID number of a first video session part of a stream currently 

6 being analyzed. 



1 22. The computer usable medium of claim 16 wherein the missing part is a packet. 
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23. The network interface of claim I, wherein the missing packets repair logical unit 
further comprises: 

a first analyzer that analyzes if a packet is missing; 

a second analyzer that analyzes which Slices are in the missing packet; and 

a repair unit that replaces the missing packets. 

24. A method for repairing missing packets in video communication over a packet-based 
network, the method comprising: 

analyzing if at least one packet is missing; 
determining which Slices are missing; 
preparing new packets which will replace the Slices; and 
sending the new packets to a destination. 

25. The method of claim 9 wherein the missing part is at least one Slice. 
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ABSTRACT OF THE INVENTION 

A network interface unit is provided with a missing packets repair logical unit, 
which repairs real time video transmissions by replacing the missing packets. The 
missing packets repair unit detects and replaces missing packets in a received video 
stream. The missing packets can be detected by finding packets that are out of sequence. 
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BACKGROUND 



Field of the Invention 

The field of the invention is generally video communication and more 
specifically improved video communication quality over a packet-based network such 
as an Ethernet or Internet Protocol (IP) using H.323 protocol or similar standard. 

Background Art 

Conventionally, a packet-based network such as an Ethernet or IP does not 
offer an end-to-end connection with a guarantee that all packets will reach their 
destination. Therefore communication over a packet-based network using different 
technologies is necessary to overcome packet loss. For example, communication that 
is not performed in real time, such as e-mail, uses protocols such as Transmission 
Control Protocol (TCP), which is a handshake protocol that verifies the arrival of all 
packets. If a packet is not received, a request is made for a retransmission of the lost 
packet. Alternatively, other protocols, such as User Datagram Protocol (UDP)/IP, 



typically send a notification about the problem of the packet loss and then handle the 
lost packet separately. 

Video communications are performed in real time. However, any delay 
between endpoints (a physical location or apparatus that can generate and/or terminate 
information streams) reduces the quality of the communication performed in real 
time. Furthermore, a handshake protocol such as TCP/IP increases the delay, which is 
the time interval between the action of one side and the response of the other side. 
The increased delay damages the flow of a conversation. Consequently, TCP/IP is a 
difficult protocol to use for real time communications. 

Therefore, video communications use UDP/IP with H.323, Session Initiation 
Protocol (SIP), or similar standards instead of using TCP/IP. UDP/IP is used 
primarily for broadcasting video streams over a network and it is a connectionless 
protocol, which runs on top of IP networks. Unlike TCP/IP, UDP/BP has very few 
error recovery services. Instead UDP/IP sends and receives datagrams over an IP 
network directly. Since UDP/IP does not have a feedback mechanism the packets that 
do not reach their destination will be lost. The percentage of packets lost during 
transmission is typically up to about 10%. 

Conventional video communications use compression standards such as 
H.261, H.263 and Moving Pictures Experts Group (MPEG). These video 
compression standards typically have two types of frames: the "Intra" frame, which is 
a non-referential image, and the "Non intra" frame, which is a referential frame, 
similar to but not limited to frames such as Inter frame (P frame), B frame, and PB 
frame. Video compression standards use the difference between the current "Non 
Intra" frame and a previous frame for compression, regardless whether the previous 
frame was an Intra or "Non Intra" frame. 
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A video packet includes video parts, which may be a whole frame or part of a 
frame. Each frame may include one or more parts, where a part of a frame may 
include, for example, a few Groups Of Blocks (GOB), part of a GOB, Slice, part of a 
Slice or a few Macro Blocks (MBs). More information about said GOBs, Slices and 
MBs can be found in standards H.261 , H.263, MPEG, etc. 

Since video compression (i.e., encoding/decoding) is based on referential 
information, "Non Intra" frames, the loss of a packet may damage video quality for all 
later frames until a new Intra frame arrives. The loss of the packet may also cause 
loss of video synchronization. Typically, a video decoder of an endpoint handles lost 
packets by requesting transmission of an Intra frame so that the referential 
information of the Infra frame can be used to correct the synchronization and be the 
basis for calculating the referential frames that follow. This request for transmission 
can be performed by using a Video Update Picture Indication (VUPI) request. 

A Multipoint Control Unit (MCU) is a node on a network, which provides the 
capability for two or more terminals to participate in a multimedia communication 
session. Therefore, the MCU may transfer a received data stream with missing 
packets, to multiple participants' endpoints. Thus the use of MCUs may create a 
bigger problem than just a lost packet in an endpoint-to-endpoint direct connection, 
because the lost packet will be missing to multiple endpoints rather than just one 
endpoint. . There are few known ways by which an MCU handles missing packets; 
none of which replace the missing packets. For example, the MCU may send video 
source packets while ignoring the missing packets, and let the endpoints make a VUPI 
request. In a videoconference with more than two endpoints, several participants may 
simultaneously receive communications from one source, such as a conference 
speaker. If a packet is lost during transmission from the video source (the conference 
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speaker) to the MCU, each of the participants would request a VUPI from the MCU. 
However, the numerous VUPI requests and the INTRA frames that the follow VUPI 
requests could overwhelm the network, especially if there is a high packet loss rate. 
Further, during Internet communications each participant may experience packet loss 
at a different time, and will therefore request a VUPI at different times, thereby 
overwhelming the network with sporadic transmissions of Intra frames. Since Intra 
frames are approximately 10 times "bigger" than "non Intra" frames, transmission of 
Intra frames may cause frame rate derogation and reduces the video quality of the 
conference. 

Therefore, there is a need for a system and method for improving video 
communications quality over a packet-based network. 
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1 SUMMARY OF THE INVENTION 

2 The present invention provides a novel method and apparatus for repairing 

3 video stream errors such as missing packets of an incoming compressed video stream. 

4 The system identifies a missing part or unit of a video stream (e.g., a lost packet) by 

5 tracking a sequence number associated with each packet. An analysis is then 

6 performed to determine which sub-units (e.g. GOBs, Slices, and/or MBs) are absent 

7 from the video stream, and to create one or more substitute video stream parts (i.e. 

8 repair packets) to compensate for absent sub-units. In an exemplary embodiment the 

9 repair packets include the minimum required video bits to keep the stream or flow of 

10 the video stream synchronized. This minimum bit requirement means that none of the 

1 1 MacroBlocks' (MBs') video data is placed or reconstructed in the packet. Instead, a 

12 decoder treats a repaired video stream part, such as a GOB, as an uncoded part that 

13 represents the same information as in the previous frame. The repaired information is 

14 then transferred to its destination and replaces the original information. 

15 Additionally, the MCU may ask for an "update request" from the video 

16 source, which may include a VUPI request. When compression standard allows, only 

17 a part of a picture that includes the missing sub-units (such as MBs) is requested. An 

18 example of a compression standard that allows for the request of a partial picture is 

19 H.263. 

20 Replacement of the missing parts provides for continuity of the video stream 

21 at a receiver site. This continuity enables smooth parsing, without synchronization 

22 losses and prevents re-synchronization process by the video "decoders." The decoder, 

23 among other places, may be located inside the MCU, or outside of the MCU at the 

24 participant's endpoint. 
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1 Alternatively, if the missing part belongs to a nonreferential image (e.g., an 

2 Intra frame), then the video stream is not corrected. Instead a new nonreferential 

3 image (such as Intra frame) is requested. 
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1 BRIEF DESCRIPTION OF THE DRAWINGS 

2 The invention will be more readily understood from reading the following 

3 specification and by reference to the accompany drawings showing an example of the 

4 invention. 

5 FIG. 1 is a block diagram of a prior art network interface unit; 

6 FIG. 2 is a block diagram of an exemplary embodiment of the present 

7 invention; and 

8 FIG. 3 is a flow diagram of the steps of an exemplary embodiment of the 

9 present invention. 
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1 DETAIL DESCRIPTION OF THE INVENTION 

2 Fig. I is a block diagram of a prior art network interface unit 100. As shown, 

3 a Local Area Network (LAN) controller 1 10 grabs required multimedia packets from 

4 a LAN 101 via a buffer 111. Preferably, the LAN controller 110 then sorts the 

5 multimedia packets into two types of streams: control streams 160 and media streams 

6 135. The control stream 160 is transferred via buffers 161 to a backplane 170. 

7 The media streams 1 35 are transferred to a Real Time Transfer Protocol (RTP) 

8 unit 120, a packetizing/depacketizing unit that sorts the streams into Audio streams 

9 150, Video streams 140 and Data streams 130 and transfers them via buffers 141, 151 

10 and 131, respectively, to the backplane 170. More specifically, video streams 140 are 

11 transferred through buffer 141 to the backplane 170 without repairing the missing 

12 packets, while audio streams 150 are transferred via buffer 151. 

13 Conversely, FIG. 2 is a block diagram illustrating an exemplary embodiment 

14 of a network interface unit 200 according to the present invention. Preferably, the 

15 network interface logical unit 200 includes a LAN controller 110, RTP unit 120, 

16 Video streams 140, Missing Packets Repair logical Unit (MPRU) 210, Packet 

17 Analyzer logical unit (PA) 220, memory 230, GOB history memory 240 ? GOB 

18 Analyzer logical unit (GA) 250, Packet Repair logical unit (PR) 260, and Temporary 

1 9 Packets Memory (TPM) 265. 

20 It should be noted that the term "GOB" represents a "Slice" for situations 

21 where slices are used instead of GOBs. In such a case, the GOB ID Number is 

22 replaced by the Slice ID, which is the MBA field in the Slice header. The MBA 

23 indicates the first MB in the Slice. 
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1 The network interface logical unit 200 is similar to network interface unit 100 

2 (FIG. 1) in that they both include a LAN controller 110 and RTP unit 120. However, 

3 in contrast to the prior art device of network interface 100, the FIG. 2 system also 

4 includes a Missing Packets Repair logical Unit (MPRU) 210: 

5 Operationally, video streams 140 from the RTP 120 are transferred to the 

6 MPRU 210. Inside the MPRU 210 the video stream 140 is distributed to three logical 

7 units: PA 220, GA 250, and PR 260. The PA 220 reads a current packet sequence 

8 number of a received packet, compares the current sequence number to the expected 

9 sequence number, and in case of mismatch sends indications of packet loss to the 

10 LAN controller 1 10, GA 250, and PR 260. The expected sequence number is based 

1 1 on the history of the received packets stored in its memory 230. 

12 The GA 250 analyzes which GOBs/Slices are associated with each received 

1 3 packet and stores this information in the GOB history memory 240. The analyzing 

14 can be done by parsing the video stream of a packet and/or by grabbing the 

15 information regarding the GOB/Sli|| ID from the RTP header. This information 

16 about the GOB/S|ik^ header in the RTP Header does not always exist and is based on 

17 the Rfj£ 

18 When the GA 250 receives an indication that there are missing packets prior to 

19 the current packet, GA 250 uses the last stored information in the memory 240 and the 

20 first GOB/Slice ID Number that was found in the current packet to analyze which 

21 GOBs/MBs are missing. The GA 250 then sends this information to the PR 260, 

22 which in turn receives the incoming video stream 140 of packets and stores them in 

23 the Temporary Packet Memory (TPM) 265. Thus, the TPM 265 receives both 

24 indications about a missing packet from the PA 220 and the identification numbers of 

25 the corresponding missing GOB/MBs from the GA 250. If there are no missing 
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1 packets, the PR 260 reads the last stored packet from the TPM 265 and sends the 

2 stored packet, as is, via buffer 271 to the backplane 170. When the PR 260 receives 

3 the indications from the PA 220 and from GA 250, the PR 260 creates repaired 

4 packets and sends the corrected stream, including the repaired packets with the stored 

5 packets from the TPM 265 via buffer 271 to the backplane 170. The determination of 

6 which GOBs/MBs are missing is based on the compression standard as explained 

7 below by example. 

8 Fig. 3 is a flow diagram illustrating steps of an exemplary method for 

9 repairing missing packets according to the present invention. Initially, the MPRU 

10 210 (FIG. 2) receives a next packet in step 300. The PA 220 (FIG. 2) then reads 

1 1 the sequence number of the packet in step 310 and compares the sequence number 

12 of the packet to the sequence number of the previous packet stored in the memory 

13 230 (FIG. 2). Subsequently, the PA 220 (FIG. 2) checks if the packet is 

14 consecutive in decision box 320 based on the comparison of the sequence 

15 numbers. If the packets are consecutive, the PA 220 stores the current packet's 

16 sequence number in the memory 230 and returns to step 300 to receive a new 

17 packet. However, if the packets are not consecutive, meaning that one or more 

18 packets have been lost, then PA 220 checks if the missing packet belongs to an 

19 Intra frame or "Non Intra" frame in step 340. 

% 20 One way of keeping track of the type of frame is to sample the indication 

21 of frame type. For example, the PA 220 may check the frame type by reading a 

22 "frame's header," GOB "0" of the last packet. In H.263, the frame header is part 

23 of a header of GOB 4t 0" while in H.261 the frame header is identified by GOB ID 

24 Number "0". 
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1 If the missing packet is part of Intra frame 340, no packet repair is necessary 

2 and a VUPI request is performed in step 380 and the process returns to step 300 to 

3 receive the next packet. If the packet contains part of a "Non Intra" frame the PA 220 

4 sends a missing packets acknowledgement to the GA 250 (FIG. 2) and the PR 260 

5 (FIG. 2). The GA 250 in step 350 uses the information stored in the memory 240 

6 (FIG. 2) to get the last GOB ID Number that arrived before the missing packets. For 

7 example, if the last GOB before the missing packet is GOB (j), then the GA 250 finds 

8 the first GOB in the current packet in step 360, e.g., GOB (k). Based on these two 

9 GOB ID Numbers (i.e., (j) and (k)), the compression standard, and the source format 

10 (e.g., Common Intermediate Format (CIF), 4 CIF, or Quarter CIF (QCIF)), the GA 

1 1 250 determines in step 365 which are the missing GOBs. 

12 In the case that the compression stream is using Slices instead of GOBs, the 

13 GA 250 analyzes and stores the last Slice ID, which is the MBA field in the Slice 

14 header of the last received packet. Then the GA finds the Slice ID of the first Slice. 

15 By using this two MB As and the width of the Slice the GA 250 analyzes which MBs 

16 are the missing. 

17 The following are a few examples of the analysis, for video algorithms H.261 
18 x and H.263 and two sources formats CIF and QCIF. The numbers in the following 
19 table specify the sequence of GOB ID Numbers used in those cases. 

20 



21 
22 





QCIF 


CIF 


H.261 


0,1,3,5 


0,1,2... 12 


H.263 


0,1,2...8 


0,1...16,17 



1 1 



1 The GA 250 recognizes the missing GOBs by finding, the GOBs missing in the 

2 sequence. Thus: 

3 a. For H.261 and QCIF, if j=0 and k=5, then the repaired packet would 

4 include GOBs 1 and 3. 

5 b. For H.261 and CIF, if j=4 and k=0, then the repaired packet would 

6 include GOBs 5,6,7... 12. 

7 c. For H.263 and QCIF, if j=6 and k=2, there are 2 packets that need to be 

8 repaired. The first packet includes GOBs 7 and 8, and the second 

9 packet includes GOBs 0 and I. A packet is not allowed to include 

10 GOBs from different frames. 

1 1 d. For H.263 and CIF, if j= 17 and k=6, the repair packet includes GOBs 

12 0,1, 2.. .5. 

13 e. For H.261 and CIF, if j=4 and k=5 no repaired packet is needed. 

14 If a GOB ID Number is not found in the current packet, no analysis can be 

1 5 performed and the GA 250 drops the current packet. Subsequently, the process 

16 returns to step 300 and waits for the next packet. 

17 The GA 250 transfers the results of its analysis (i.e., the missing GOB ID 

18 Number) to logical unit PR 260 (FIG. 2), which in turn prepares the necessary 

1 9 repaired packet or packets in step 370. 

20 The packet repair is done based on the compression standard, and the 

21 source format, each packet includes the necessary bits plus some extra bits or 

22 "stuffing" that keep the continuity of the stream as required by the protocol in use. 

23 For example H.261 requires just the GOB headers with no byte alignment, while 

24 H.263 requires 1 bit for each non-coded Macro Block (MB). Since H.263 CIF 

25 GOB includes 22 MBs, 22 bits of T are needed. H.263 QCIF GOB includes only 
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1 1 1 MBs, so in this case 1 1 bits of * P are needed. These bits are added to the GOB 

2 header. Further, the H.263 frame should be byte aligned, which is guaranteed by 

3 the H.263 stuffing. 

4 The PR 260 sends the repaired packet or packets with the current packet as 

5 a corrected video stream to the backplane in step 390, and sends an update request 

6 to the video source in step 380. Conversely, if the PR 260 does not get a missing 

7 packet indication from the PA 220, the PR 260 sends the current packet, as is, to 

8 buffer 27 1 . The update request might be a VUPI request, however it is not always 

9 necessary. Alternatively, in H.263 a message "VideoNotDecodedMb" may be 

10 used and only the missing GOBs/MBs may be updated. 

1 1 The MPRU 210 can be implemented by software (e.g., an additional software 

12 package loaded to an existing processor, such as RTP processor 120). Alternatively, 

13 the MPRU 210 may be an additional processor including programs or special 

14 hardware for all three' units, the PA 220, the GA 250 and the PR 260, or three separate 

15 processors, one per unit or any combination of one to three processors and/or 

16 hardware units. Memories 230, 240 and TPM 265 (FIG. 2) can be implemented by 

17 any one of, or any combination of, SRAM, DRAM, SDRAM, internal and/or external 

18 memory. 

19 Although the above exemplary embodiments are given in terms of streams, 

20 packets, GOBs, and Slices, the invention can be applied to video streams that are 

21 divided into other units, parts, and/or sub-units. Similarly, the missing packet repair 

22 unit, MPRU 2 10, is just an example of a missing part repair unit. 

23 As can be understood from the above description, the present system improves 

24 traffic over a network by replacing missing GOBs/Slices, and thereby enhances the 

25 continuity of data streams at a receiver side. The continuity of the data stream enables 

13 



1 smooth parsing (without synchronization losses and re-synchronization process) by 

2 video "decoders'* and improves the quality of video communication. In the 

3 description and claims of the present application, each of the verbs, "comprise" 

4 "include" and "have", and conjugates thereof, are used to indicate that the object or 

5 objects of the verb are not necessarily a complete listing of members, components, 

6 elements or parts of the subject or subjects of the verb. 

7 While the invention has been described with reference to specific 

8 embodiments, it will be understood by those skilled in the art that various changes 

9 may be made and equivalents may be substituted for elements thereof without 

10 departing from the true spirit and scope of the invention. In addition, modifications 

1 1 may be made without departing from the essential teachings of the invention. 
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WHAT IS CLAIMED 

1 1 . An apparatus for facilitating multimedia communication between a plurality of 

2 endpoints over a packet based network, each respective endpoint sending a 

3 compressed video output signal and receiving a compressed video input signal, having 

4 at least one network interface to a packet-based network, the network interface 

5 comprising: 

6 a missing packets repair logical unit, said missing packets repair logical unit 

7 handling missing packets and thereby maintaining continuity of a video stream and 

8 reducing traffic over said network. 

1 2. The network interface of claim 1, wherein the missing packets repair logical unit 

2 further comprises: 

3 a first analyzer that analyzes if a packet is missing; 

4 a second analyzer that analyzes which Group Of Blocks (GOBs) are in the 

5 missing packet; and 

6 a repair unit that replaces the missing packets. 

I 3. The apparatus of claim 1 wherein the network is a local area network. 

1 4. The apparatus of claim 1 wherein the network is a wide area network. 

1 5. The apparatus of claim 1 wherein the video stream is repaired during a video 

2 stream receiving. 
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1 6. A system comprising: 

2 a network interface unit having 

3 a video stream repair unit that receives a video stream from a real time 

4 protocol unit, having 

5 a detector unit that detects missing packets, 

6 an analyzer unit that analyzes which video parts are missing, 

7 a replacement unit that 

8 receives 

9 an indication from the detector unit that packets are 

10 missing, and 

1 1 information from the analyzer unit, the information 

12 including which video parts are missing, and 

13 in response to receiving the indication and the information 

14 replaces a missing packets in a video stream during transmission of 

15 the video stream over a network as part of a receive process. 
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1 7. A method for repairing missing packets in video communication over a packet-based 

2 network, the method comprising: 

3 analyzing if at least one packet is missing; 

4 determining which Group of Blocks (GOBs) are missing; 

5 preparing new packets which will replace the GOBs; and 

6 sending the new packets to a destination. 

1 8. The method of claim 7 wherein the destination is a point remote from where the 

2 analyzing occurs. 

1 9. A method comprising: 

2 transmitting a video session over a network; and 

3 replacing a missing part from the video session while the video session is being 

4 transmitted. 

1 10. The method of claim 9 wherein the missing part is replaced in an intermediate node 

2 during a receiving process that receives the video session. 

1 11. The method of claim 10 further comprising detecting the missing part during the 

2 receiving process. 
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1 12. The method of claim 1 1 wherein detecting the missing part includes detecting if a 

2 part of the video session received is out of sequence 

1 13. The method of claim 9 wherein replacing the missing part is dependent on the frame 

2 type. 

1 14. The method of claim 12 wherein: 

2 each part of a video session has an ID number; and 

3 analyzing which part is missing comprises 

4 finding the ID number of the last received video session part , and 

5 finding the ID number of a first video session part of a stream currently 

6 being analyzed. 

1 15. The method of claim 9 wherein the missing part is at least one GOB. 

1 16. A computer usable storage medium having stored thereon a method comprising: 

2 transmitting a video session over a network; and 

3 replacing a missing part from the video session while the video session is being 

4 transmitted. 
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17. The computer usable storage medium of claim 16 wherein the missing part is 
replaced in an intermediate node during a receiving process that receives the video 
session. 

18. The computer usable storage medium of claim 17, the method further comprising 
detecting the missing part during the receiving process. 

19. The computer usable storage medium of claim 18 wherein detecting the missing part 
includes detecting if a part of the video session received is out of sequence 

20. The computer usable storage medium of claim 16 wherein replacing the missing part 
is dependant on a frame type. 

21 . The computer usable storage medium of claim 19 wherein: 

each part of a video session has an ID number; and 

analyzing which part is missing comprises; 

finding the ID number of the last received video session part , and 
finding the ID number of a first video session part of a stream currently 

being analyzed. 



19 



7 



I 22. The computer usable medium of claim 16 wherein the missing part is a packet. 

1 23. The network interface of claim 1, wherein the missing packets repair logical unit 

2 further comprises: 

3 a first analyzer that analyzes if a packet is missing; 

4 a second analyzer that analyzes which Slices are in the missing packet; and 

5 a repair unit that replaces the missing packets. 

1 24. A method for repairing missing packets in video communication over a packet-based 

2 network, the method comprising: 

3 analyzing if at least one packet is missing; 

4 determining which Slices are missing; 

5 preparing new packets which will replace the Slices; and 

6 sending the new packets to a destination. 

1 25, The method of claim 9 wherein the missing part is at least one Slice. 
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ABSTRACT OF THE INVENTION 

A network interface unit is provided with a missing packets repair logical unit, 
which repairs real time video transmissions by replacing the missing packets. The 
missing packets repair unit detects and replaces missing packets in a received video 
stream. The missing packets can be detected by finding packets that are out of sequence. 
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